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The RICIS Concept 


The University of Houston -Clear Lake established the Research Institute for 
Computing and Information Systems (RICIS) in 1986 to encourage the NASA 
Johnson Space Center (JSC) and local industry to actively support research 
in the computing and information sciences. As part of this endeavor* UHCL 
proposed a partnership with JSC to jointly define and manage an integrated 
program of research in advanced data processing technology needed for JSC’s 
main missions, including administrative, engineering and science responsi- 
bilities. JSC agreed and entered Into a continuing cooperative agreement 
with UHCL beginning in May 1986, to jointly plan and execute such research 
through RICIS. Additionally, under Cooperative Agreement NCC 9-16, 
computing and educational facilities are shared by the two institutions to 
conduct the research. 

The UHCL/RICIS mission is to conduct, coordinate, and disseminate research 
and professional level education in computing and information systems to 
serve the needs of the government, industiy, community and academia. 
RICIS combines resources of UHCL and its gateway affiliates to research and 
develop materials, prototypes and publications on topics of mutual interest 
to its sponsors and researchers. Within UHCL, the mission is being 
implemented through interdisciplinary involvement of faculty and students 
from each of the four schools: Business and Public Administration, Educa- 
tion, Human Sciences and Humanities, and Natural and Applied Sciences. 
RICIS also collaborates with industry in a companion program. This program 
is focused on serving the research and advanced development needs of 
industry. 

Moreover, UHCL established relationships with other universities and re- 
search organizations, having common research interests, to provide addi- 
tional sources of expertise to conduct needed research. For example, UHCL 
has entered into a special partnership with Texas A&M University to help 
oversee RICIS research ani education programs, while other research 
organizations are involved via the “gateway" concept 

A major role of RICIS then is to find the best match of sponsors, researchers 
and research objectives to advance knowledge in the computing and informa- 
tion sciences. RICIS, working jointly with its sponsors, advises on research 
needs, recommends principals for conducting the research, provides tech- 
nical and administrative support to coordinate the research and integrates 
technical results into the goals of UHCL, NASA/JSC and industiy. 
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1) ABSTRACT: 

This paper reports an extension of the MIAG algorithm for recognition and motion parameter deter- 
mination of general 3D polyhedral objects based on model matching techniques and using Moment Invari- 
ants as features of object representation. Results of tests conducted on the algorithm under conditions simu- 
lating space conditions are presented. 

2) INTRODUCTION: 

Many different object recognition and attitude determination techniques have been proposed by 
researchers. The earliest ones used the approach of matching the observed image to a library of a fixed 
number of views of objects. The limitations of such an approach are glaringly apparent. Among the later 
techniques, Richard and Hemami [l] used Fourier descriptors and Dudani et al [2] used moment invariants. 
Watson and Shapiro [3] used a model matching technique to identify wireframe perspective views. Their 
method is iterative and requires use of a numerical optimization technique. Marr and Poggio [4] have 
implemented a stereo reconstruction algorithm which uses geometric constraints to recover surface shape. 
Similar range data techniques have been developed by other researchers also. A fundamental limitation of 
these techniques is the introduction of restrictive assumptions about the imaged scene in terms of general- 
ized cones [5] or in terms of planar and quadric patches [6]. Horn [7] has worked on the extraction of shape 
from shading, using the reflectance map. This method uses the brightness gradient as the image feature 
used in recovery. It is applicable to smooth, uniform Lambertian surfaces. Stevens [8], Render [9] and later 
Witkin [10] have tried to recover shape from texture. This technique and also the shape from contour (sur- 
face boundaries) technique presented by Bairow and Tenenbaum [11] rely on the assumption that the world 
of objects is regular. Such techniques are limited to smooth-textured surfaces. For some other contributions 
see Silcox [12]. 

Bamieh and deFigueiredo [12] have developed the Moment-Invariants / Attributed Graph (MIAG) 
Algorithm in which 2D moment invariants which are invariant under 3D motion, have been used for the 
recognition of 3D objects, using an attributed graph representation and based on the concept of model 
matching. This approach avoids restrictive geometric assumptions and so offers an advantage over most 
techniques discussed above. In its original form this algorithm was applicable for recognition of 
polyhedral objects but it could not be used for attitude determination if the polyhedron had symmetric faces 
for reasons discussed later in this paper. This limitation h as been overcome now as discussed in this paper. 
As this technique uses moment invariants as Features of representation and these can be computed only for 
planar faces* the technique is not direcdy applicable to the recognition and attitude determination of curved 
objects. However we can use other representational features such as the Gaussian and mean curvature and 
the attributed graph and model matching techniques can still be applied. 

To implement this technique we need picture information in the form of wireframes. So the picture 
from the camera is digitized and converted to wireframe form before applying the MIAG algorithm to it. In 
the work currently in progress, the overall process is divided into three parts: 
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1) Data acquisition and digitization. 

2) Wireframe extraction. 

3) Recognition and motion parameter determination. 

The first two parts above constitute the image processing/feature extraction stage. The work in this 
stage is briefly outlined below. 

3) IMAGE PROCESSING / FEATURE EXTRACTION: 

3.1) Data acquisition and digitization: Models of various space objects, such as mockups of satel- 
lites, the space shuttle and parts of the space station are being used to test the performance of the algorithm. 
These models are grabbed by cameras under illumination conditions simulating those prevelent in space 
orbit. The pictures are digitized to obtain 2D arrays of brightness values. This is the initial level of 
representation in the system. 

3.2) Wireframe extraction: Wireframe extraction consists of removing noise from the picture and 
subjecting it to edge detection and reconstruction. The input image is lowpass filtered to remove the high 
frequency noise. A 7x7 Gaussian filter is used for this. The Sobel gradient operator is then applied to the 
output of the lowpass filter to obtain an edge detected version of the input image. This image is a grey level 
image. It is converted to binary form by thresholding, which also removes some of the noise and thins 
down the edges. The remaining noise is removed by median filtering. A length 5 filter was employed for 
this. The output so obtained is a noise free, binary edge image. But the edges are thick smears instead of 
the fine lines required in a wireframe. A thinning algorithm [13] is applied to this to reduce the edges to 
unit pixel thickness, thus obtaining the required wireframe. 

4) RECOGNITION AND MOTION PARAMETER DETERMINATION: 

The MIAG algorithm [12] is an algorithm of recognition and attitude determination of 3D objects. 
We discuss this algorithm in two steps: First, object recognition and second, attitude determination, 

4.1) Object recognition: The MIAG technique recognizes a 3D object from its projection on an 
imaging plane. The algorithm works for the identification of polyhedral objects. Each face of a polyhedron 
can be considered as a rigid planar patch (RPP). Motion of the object can then be considered as motion of 
its constituent RPP’s. If it is assumed that the image is formed by parallel projection then if an RPP under- 
goes rigid body motion in 3D its image undergoes affine transformations. So the method which tries to 
identify an object in 3D motion should use features of images which remain invariant under affine transfor- 
mations. General moment invariants are such features. They remain invariant under translation, rotation 
and scale changing. Moments are coefficients in a series expansion of the image function, similar to those 
in a Fourier series expansion. But unlike in Fourier series where sine and cosine functions are the basis 
functions, here the basis functions are polynomials in the linage function variables. Thus if the picture 
function is f(x,y) its moment is: 

m pq- J J * p y q f{x y y)dxdy 
for p,q=0, l,2.i. -• ’ 

The value of (p+q) is known as the order of the moment. Theoretically, for a perfect description of 
the picture in terms of moments, p and q should go to °° . But in the present algorithm moments upto only 
order four have been used. This is because the computation of higher order moments is increasingly 
difficult and it was found fiiat picture representation in terms of four moments gives good results in the test 
cases. 

These moments have to be computed for each face of the picture wireframe. The picture intensity is 
taken to be 1 inside a polygon and 0 outside it. Thus all the picture information is contained in its boun- 
daries. Using this fact, the above surface integral can be changed to a line integral by Green’s theorem. For 
a digital picture the integral reduces to a sum. See [12] for details. 

Moment invariants of all faces of certain standard objects are stored in the system library. Given IT 
wireframe which needs to be identified, the moment invariants of each of its faces are computed. These are 
then matched to the stored values of the moment invariants of the library objects. If all the moments 
corresponding to a face of the RPP match all the moments of a face of a stored object, we can say that the 
two faces are similar. For the objects to be the same, not only should the faces be similar but the adjacency 
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conditions of the faces and the angles between the faces should be similar. To carry out this matching, the 
wireframe is first converted to an attributed graph, Each node of the graph represents a face of the 
wireframe. If two nodes are connected by a line (edge) it means that the faces corresponding to these nodes 
are adjacent. With each node is associated a feature vector consisting of a set of moment invariants of the 
face that it represents and with each edge is associated a scalar which gives the angle between normals to 
the two faces it connects. As an example, the attributed graph representation of a cube is shown in Fig 1 . 

The algorithm works as follows: Suppose we hypothesize that node Wj in the wireframe corresponds 
to node 0 y in the model graph. If W f has k nodes adjacent to it, W mU .. then 0 ; should also have k 
nodes adjacent to it, The following constraints should be satisfied: 

1) W w must have the same feature vectors as O vt s=l,...,k. 

2) Angle between and should equal angle between 0 „ and Oj for all s=l,...,k. 

3) If any two W w ’s are connected then the angle between them should equal that between their 
matching nodes. 

If all these conditions are satisfied, then an admissible matching configuration is said to have been 
obtained at nodes W ; and O y If matching configurations are obtained between all nodes of the given 
wireframe and one of the stored models, then we can say that the wireframe matches the model. In most 
cases, only a small part of the given model needs to be matched to discriminate it against the other models 
in the library. 

It may be noted that because of numerical truncations and rounding during calculations there may 
not be a perfect match between the moment invariants computed for the wireframe and those stored for the 
model. So we define a measure of error between the two sets of moment invariants. The moment invariants 
can be taken as coordinates of a point in four dimensioinal vector space and the distance between the two 
points is taken as a measure of error. If / 2 , / 3 and / 4 are the moment invariants of a wireframe’s face and 
/'i, V 2 , /' 3 and / 4 those of a model’s face then the distance is: 

d- V(/, - / i ) 2 Pi 2 + (h ~ + (^3 “ + (A* - 


where the p’s are weighting factors. These are needed to equalize the contribution of all four moment 
invariants in the error measure because some of the moment invariants may have values of the order of 
10" 3 and others may be of the order of KT 7 . If the value ’d’ is less than a certain threshold (taken 0.01 
here), the two sets of moment invariants are taken to be equivalent. 

The driver algorithm arbitrarily picks a node W J in the wireframe, then it looks for a node 0 } in the 
model with the same feature vector. If matched, these nodes are marked as a pair. An adjacent image face 
is chosen and the adjacent object faces are scanned to see if one of them matches it. As each adjacent pair 
is found it is checked for consistent adjacency and equality of angles between faces. If everything matches 
satisfactorily a succesful match is declared. 


4.2) Attitude determination: The identity of the object having been so determined, one has to esti- 
mate the attitude and location of the recognized object relative to a library standard. 


Let (X,Y ,Z) be the original coordinates of a point on the body and (X’.Y’^Z*) be its coordinates after 
motion. Then, 


r 


r 

T 


= R 


X 

Y 

Z 


+ r 


where 


R = 


r\ r 2 r 3 

'4 '5 r 6 
>7 r % r % 


is the rotation matrix and 
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r= 


'ax' 

AY 
A Z 


is the translation matrix. 


Let the corresponding points on the image be (x,y) and (x’,y’). Then, 


V 


r^x+r^ryZ+Ax 

/ 

y _ 


r+x+ryy+r^+Ay 


For simplicity let Z=0 i.e. the RPP in library lies in the x-y plane of the object space. Then the above 
matrix can be repressented as: 


x' 


Ql Qi 

a 


AX 

/ 


Ql Ql 

3 

+ 

AY 


where 

Ql -ri.jSi =r 2 ,Q} -r 4 ,Qi = r 5 , A x = A X . A y = A Y . 


All Q’s and A’s can be determined from the moments of the images. To find the rest of the r’s we use 
the fact that the sum of squares of any row or column in the r matrix is L 

From the r’s we can find directional cosines of the rotation axis and the rotation angle as: 


sin0 » and cosG = 
2 


~ Oj, - r \ ) 
d 1 


where 


d = V( r 8 ~ r t>f + (rj - T if + ( r 4 ~ rj 1 


(both sin 0 and cos 0 are needed to determine 0 uniquely). The direction cosines are: 

nl « (r g -r 6 )/d 

n2 = ( r ^ - r* 7) / d 
n3 = ( r 4 - r 2 ) / d 

where d 2 = (r8-r6) 2 + (r3-r7) 2 + (r4-r2) 2 

We can also find the translation in x and y directions as: 

A x = m 10 / moo ; A y « moi / "loo 


A z is not computable by this method. Also, this method cannot give rotational information for 
objects which have an axis of reflection symmetry (e.g. parallelograms, triangles) as the tensors all go to 
zero in such cases. So given an RPP whose attitude has to be determined we need to: 

a) Check whether any axis of reflection symmetry exists. 

b) Check whether it will have any axis of reflection symmetry under any affine transformation. 

c) If the face has any axis of reflection symmetry or will have it under affine transformations, subject 
it to distortion which removes the axes of reflection symmetry. 

The procedures for these steps are as follows: 

a) Symmetry conditions for polygons: To check a given polygon for axes of reflection symmetry, we 
use the concept of the Voronoi diagram . 

4,2.1) Voronoi diagram: Given a set of N points corresponding to the vertices of the polygon, let x l 
and x J be two of the points. Let P( x i , x ] ) be the half plane containing x l that is defined by the perpendicu- 
lar bisector of x 1 x J . The intersection of N * 1 such half planes, denoted by V(i) is called the Voronoi 
polygon associated with p 1 . Note that the polygons are unbounded. For N points there are N such polygons 
which partition the plane into a net called the Voronoi diagram . The construction of the Voronoi diagram 
for a pentagon is shown in Fig. 2. 
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Let the vertices of a polygon be x ] , x 2 , ... f x N where 


The points such that 


x = 


y i 


v*> = 


2 


will then be the extremities of the Voronoi diagram. 

Given a polygon its Voronoi diagram is constructed and the points v tJ obtained. For high complexity 
polygons it is usually computationally more efficient to use this procedure than to directly evaluate v ,; from 
the x l and x j . The symmetry conditions are then determined as explained below. 

The symmetry conditions for a polygon depend on whether it has an odd or even number of vertices. 

(i) If the number of vertices is odd, the axis of symmetry should pass through a vertex and the mid- 
point of two other vertices. 

(ii) If the number of vertices is even, the axis of symmetry passes through two vertices or two mid- 
points. 

For an odd polygon, an axis of symmetry to exist and pass through a point x\ the required condition 
is that there exist a set of x l and x j (i * k, j * k) such that 

<x l - x* t v l j-x k > = 0 


such x l and x> will form pairs of points symmetric with respect to the axis through x k . 

If there is to be no axis of symmetry through x\ then 

<x l - x\v k * - x k > *0 

for i * k, j * k. This is a necessary and sufficient condition to guarantee the nonexistence of any axis of 
symmetry through x k . The same procedure is repeated for ail vertices to verify whether the polygon has 
any axis of reflection symmetry. 

For an even vertex polygon, two different kinds of axes of symmetry can exist. 

i) Axis passing through two vertices. 

ii) Axis passing through two midpoints of vertices. 

i) Let x a and x b be the vertices to be checked. An axis of symmetry will pass through these vertices if 
there exists a set of x l and x J such that 

<x'-x j y j -x a > - 0 
and 

<x l - x b > = 0 

for i * a or b, j * a or b. Such x l and x * points will form pairs which are symmetric with respect to the axis 
joining x a and x b . If the line joining x a and x b is not to be an axis of symmetry then 

<x l - - x a > *0 


or 

for i * a or b, j * a or b. This is a necessary and sufficient condition to guarantee the nonexistence of any 
axis of symmetry through x a and x b . This procedure is repeated for all vertices to verify whether the 
polygon has any axis of symmetry. 

ii) Let v w and v" be the vertex midpoints to be checked. An axis of symmetry will pass through 
these if there exists a set of x l and x } such that 
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and 


< jc ‘ - x } y } - v pq > = 0 


<x i -x J y J -v r5 > = o 

for i * p, q, r or s and j *p, q, r or s. Such x l and x ' will form pairs of points symmetric with respect to the 
axis joining v pq and v". 

If the line joining v w and v" is not to be an axis of symmetry then 

<x l - x j y } - v pq > * 0 


or 

<x*-xW J -v n >* 0 

for i * p, q, r or s and j * p, q, r or s. This is a necessary and sufficient condition to guarantee the nonex- 
istence of any axis of symmetry through v pq and v* . This procedure is repeated for all combinations of 
vertices to check that no axis of symmetry exists. 

b) Having verified that no axis of symmetry exists for a polygon, we need to further verify whether 
any axis of symmetry will exist under any affine transformation of the face. 

The conditions derived above for no axis of symmetry to exist are of the general form 

uW* o 

where U and V are two dimensional vectors. Under an affine transformation A, the condition becomes 

U T A T A Vf 0 

whether this condition will be satisfied or not depends on the nature of U and V i.e. on the nature of the 
polygon and also on what kind of affine transformation it is subjected to i.e. on A. Thus a nonsymmetric 
triangle can be affine transformed to an equilateral or isosceles triangle which has axis of symmetry. 

c) If the face has an axis of symmetry as verified in a) or b) then it is subjected to distortion which 
removes the axes of symmetry. It has been found that while for any particular distortion there always exists 
an affine transformation that would yield an axis of symmetry, if the polygon is subjected to two separate 
distortions which are antisymmetric with respect to each other, there exists no affine transformation which 
yields axes of symmetry for both cases. If these two distortions are referred to as £>, and D 2 , then the pro- 
cedure consists of subjecting the polygon toZ>, and checking it according to a) and b) to see whether it has 
any axis of symmetry. If it does it is subjected to D 2 and by the above argument it will not have any axis of 
symmetry. 

It has been found that polygons with three or four vertices can always be affine transformed to sym- 
metric polygons. So a minimum of five points are needed to obtain a nonsymmetric polygon. A technique 
has been developed whereby it is not necessary for all the five points to physically lie on the polygon. If we 
have three points on the polygon, the other two points can be obtained as functions of coordinates of these 
three points. This is shown for a triangle in Fig.3, where P x , P 2 and P 2 are points on the triangle (its ver- 
tices) and Q l and Q 2 are artificially created points. The five points should be positioned such that the 
polygon formed by them is the distortion D, or D 2 referred to above. Two such distortions are shown in 
Fig.3 b and c. 

4.3) Experimental Results: Fig.4 shows certain objects that have been used to test the simulation 
of the MIAG algorithm. Fig.5 shows the output of the program for recognition and attitude determination 
of an object under two different orientations. Fig.6, 7 and 8 refer to certain physical objects that have been 
used to test the MIAG algorithm. These objects are a simple polyhedral structure, a space shuttle model 
and a space station model. These figures show these objects and their thinned wireframes. 

5) CONC LUSIO N: 

The MIAG algorithm has been extended for die attitude determination of general polyhedral objects. 
The algorithm has been tested under conditions simulating space conditions and the results are presented in 
this paper. Work is in progress to extend the algorithm to the general case of recognising and localising any 
general object. 
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Space Shuttle 


I-Beam 




T-Beam 


L-Bcnm 



Cube 


Fig.4 Examples of objects used to test the MIAG algorithm 






Fig.5 Examples of Recognition and Attitude Determination using MIAG Algorithm 



roll = 45 
pitch = 45 
yaw = 45 

Matched to cube 

Match correspondences are: 

Wireframe face # > Model face # 

0 > 0 

1 > 1 

2 > 4 



roll * 90 
pitch * 30 
yaw *- 90 

Matched to cube 

Match correspondences are: 

Wireframe face # > Model face I 

0 > 0 

1 > 1 

2 > 5 
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Object — > Edge Detector Output — > Edge Thinning Output 










Fig-8 Part of Space Station 
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ROBOTIC VISION/SENSING FOR SPACE APPLICATIONS + 
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Abtfract 

In the past decade, vision/senaing systems techniques have 
received significant anenrion for applicabons in robotics and auto- 
mation, The thrust of automation and robotics for space applica- 
tions has been proposed for increased productivity, improved relia- 
bility , increased flexibility, higher safety, and for the performance of 
tasks unsuited to humans, These benefits can be insured through 
automating time -consuming tasks, increasing 

ptoducrivity/performAnce of czew-accomplished tasks, and perform- 
ing tasks beyond the capability of the oew, This paper provides a 
review of efforts currently in progress at the NASA/ Johnson Space 
Cente r and at Rice University, the accomplishments to date, and 
some of the anticipated future developments. Both s y stem s and 
algorithms are discussed The evolution of future vision/ sensing is 
projected to include the fusion of muirisenson ranging from 
microwave to optical with multi mode capability to indude p ositio n, 
attitude, recognition, and morion parameters The algorithms for 
information extraction would incorporate aspects of intelli gence and 
knowledge for the interpolation and extrapolation of the needed 
dam. The key features of the overall system design will be m*S\ 
size and weight, fast signal processing, robust algorithms, and accu- 
rate parameter determination. These aspects of vision/senaing will 
also be discussed in this paper. 


/. Introduction 

The major space program for the next decade for the United 
States is the NASA permanently m ann e d Space Station. The Space 
Station will be a multi-purpose facility with mi— jpg which will 
indude science and applications, observation, technology develop- 
ment and demo nst r ati on, co mmer cial laboratories and pro duction 
fa dli ri et, and operational activities inch as 
repair of satellites, support of u nmann ed platforms, assembly of 
large space systems, and as a trantpormrion node for transfer to 
other orbits or planetary missioni. The importance of automation 
and robotics (AAR) for the establishment and operation of the 
Space Station hat been stressed by the Senate Appropriations Com- 
mittee (SAC). During 1984, SAC required that NASA establish an 
Advanced Technology Advisory Committee (ATAC) to identify 
advanced automation and robotics technologies for the Space Sta- 
tion program. These directives reflected the view of Congress that 
the Space Station program should not only i n c or po ra te advanced 
AAR, but should also use this opportunity to stimulate national 
development of these technologies. 


^ Some of the work reported here was su pported by NASA contract 
NAS 9-17145, the NSF grant DCR4318514, md the ONR contract 
N0001445-K-015Z 


The use of AAR for the Space Station program can be viewed 
in two major areas: teles^erotedl robotic systems for servicing, 
mainte nance, repain, and assembly; and computerised systems to 
reduce the manpower requirements of planning, monitoring, diag- 
nosis, tnd fault recovery of space systems/subsystetns. The overall 
throat is to increase the productivity through autonomy, increased 
operational capability, and flexibility. The 1985 ATAC report [1] 
contains a comprehensive s umm a ry of the goals that should be pur- 
sued for the initial station (Table 1) and the mature station, pro- 
jected for year 20 10 (Table 2). 

The dominant aspects of the technology for AAR are that it be 
lightweight, small in size, highly reliable, and multifunction. Soft 
docking with the Space Station can provide an example of the 
advanced technology needed for robotic control [2]. The station is 
likely to be flexible and susceptible to oscillations through docking 
forces. Those experimental or operational systems/processes requir- 
ing a high level of stability would be severely impacted Hence, 
c o nt ro lled robotic berthing systems and structures will become 

A amplified schematic which shows the functional elements 
of an automated system is presented in Figure 1. One of the key 
d em e n ts of the sysrem is retiring and perception. Its primary 
fonction is to provide information regarding the position of the 
object in in environment relative to the system's effector. This 
function involves isolation, description, identification, location, and 
data transmission In a broader c o n t ext s class of object properties 
which include g eo m e t ric, mechanical, material, optical, acoustic, 
electric, magnetic, radioactive, chemical, tad weight may be 
nee d e d The type, volume, and predskm/accuracy of the data 
B ea d ed wiD depend on the nature of the task to be accomplished 

It should be poimed out that vision » a tool that has broad 
applications extending beyond roboricriauromaboo. Vision is 
needed in fixed l o c a ti ons to keep track of location of objects, to pick 
and place m a rhine a/ sysa ms, rod to monitor performance of nooau- 
tomaric systems. In d e ed, vision provides intelligence and flexibility 
in aiithi Bi HiH md aoaanomased machin e s 

The evolution of virion systems seems to have been dictated 
by the need for quality inspections and productivity improvement 
Early vision systems were cameras whose data was fed to a com- 
puter. Algorithms in the host proceooricomputer determined the 
nece ssa ry scene/object parameters which were needed for controll- 
ers for various actions. In the 1970*1 this approach to vision was 
successful in many industrial applications; however, it was too 
slow, bulky, and expensive. As the role of robotics began co be 
widespread, the need for vision systems which are smaller, fester, 
more flexible, more intelligent, provide accurate images, have high 
resolution, sense color, and provide location and orie ntat ion data 
became obvious. In the latest versions, intelligence is another 
feature added to the list 
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Table 1. Proposed Coals for Automation and 
Robotic! Application!, Initial Space Station 


Automation 


o Electrical Power 

Controller* enhanced by expert ayecema for 

- Load diatribution and twitching 

- Solar array orientation 

- Trend analyaia 

- Fault diagnoti* 

o Guidance, Navigation, and Control 

Expert system for 

- Station attitude control 

- Experiment pointing 

” Orbital maintenance and reboost 

- Rendezvous navigation 

- Fault diagnosis 

o Ccwunication and Tracking 

An executive enhanced by expert systems for 

- Communication achaduling 

- Rendezvous tracking 
* Data rate selection 

- Antenna pointing 


o Information and Data Management 

An executive enhanced by expert ayateos for 
control of 

- Subsystem atatuslng 

- Trend analysis 

- Fault diagnoala 

- Redundancy and configuration management 

- Data base management 

o Environmental Control and Life Support 

Controllers enhanced by expert systems for 

- Trend analysis 
* Fault diagnosis 

- Crew alarm 

- Station atmosphere monitoring and control 

- Hyperbaric chambar 

Robotics 

Teleoperation of mobile remote manipulator with 
collision avoidance 

Mobile multiple-arm robot with dextrous manipulators 
to inspect end exchange orbital raplaceable units 

Systems designed to be serviced, maintained, end 
repaired by robots 


Table 2. Proposed Goals for Advanced Automation 
and Robotics Applications, Mature Space Station (2010) 


Automation 


o Propulsion 

An intelligent controller for 

- Fuel distribution and management 
“ Leak detection and evaluation 

o Electrical Power 

An autonomous intelligent controller for 

- Power management 

- Fault da taction and isolation 
” Maintenance scheduling 

o Guidance, navigation, and Control 

An intelligent controller for 
“ Fully automatic rendezvous and docking 

- Spaca traffic control 

“ Remotely piloted vehicles 

- Collision avoidance 

o Communication and Tracking 

An intelligent system for 

- Automatic planning 

- Tracking multiple - vehicles 

* Scheduling bulk data storage for coMunl- 
cations blackouts 

- Detection, identification, and character- 
lzatlon of general targets 

o Information and Data Management 

An intelligent system for 

- Fault detection, isolation, and repair 
" Natural language interface with crew 

A data base manager for 

- CAD/CAE bulk data storage facility 

- Retrieval and routing to requestors 


o Environmental Control and Life Support 

An intelligent controller for 
" Ensuring fail-eaf«/f ail-operational modes 

- Fault detection and isolation 

~ Chemical analysis of air and water 

- Toxic gas analysis 

o Habitability 

An intelligent system for 

- Health maintenance 

- Speech interpretation and synthesis 

- Physiological monitoring 

- Automated medical decisions 

- Trend analysis 

o Structures and Mechanisms 

Advanced work station 
Intelligent actuators 
Teleoperators 

o Orbiting PJUitfonu 

An intelligent system for 

- Process control 

- Maintenance control 

- Planning and trend analysis 

Robotics 

Inspection, maintenance, refurbishment, and repair 
Fuel end meterlale trenefer 
Detecting hazardous leeks 
Satellite rttrlevel end eervlcing 
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A* the new era in robotics dawns, the 3D scene/object descrip- 
tion, along with location, orientation, and morion parameters, will 
be needed Multisensor capability is deemed essential in order to 
have the data for the determination of these parameters. Sensor 
complements may include both microwave and optical with diverse 
modes. The fusion of these sensors for the mensuration of various 
parameters in an efficient manner provides by far the greatest chal- 
lenge for the organization and use of partial or noisy information in 
a coherent m a nn er. The compression, storage, and transmission of 
the information associated with multisensor capability require novel 
algorithms. Finally, both in the tingle* and multi pie -sensor cases, 
the need for a dramatic increase in processing speed to achieve 
real-time vision sensing goals will constitute an important con- 
sideration in hardware design and development Current efforts to 
develop highly parallel multiprocessor arrays, pipelines, and other 
appropriate network configurations for this purpoose represent a 
step in this direction. 

In to p«per, the requirements for data are viewed from the 
applications standpoint A review of the systems aspects of 
vition/seosing is provided. The progress in the area of algorithm 
development is summarized. Additionally, new concepts in the 
areas of sensor and algorithm development are elaborated. 

JJ. Genera/ Requirements for Space Vision l Sensing Applica- 
tions 

One of the first functions that has been planned for the Space 
Station is the assembly itself. This can involve mating structures, 
bolting, lo c king , and forming joints in the structure itself. An 
option for the assembly of the structure is to use of Shuttle Remote 
Manipulator System (RMS), controlled either by astronauts during 
Extra-Vehicular Activity (EVA) or toe in the cabin (Figure 2). 
The initial vision system could be remote TV cameras with the 
prime function of providing imagery for inspecdoa As time 
proceeds, the awembly processes could involve Orbital Maneuver- 
ing Vehicle (OMV) with robotic arms and season for vision. The 
parts assembly could involve resistance spot welding, electron beam 
welding for continuous joint and seal formation, localized light 
forming, and precision light machining. Such functions will need 
dramatic improvement in sensing and perception to achieve accurate 
mechanical control and monitor the performance of the OMV. Ttm 
versatility of foe OMV can be Increased by interchanging special- 
ized lightweight end -effectors [2]. Ultimately, in to distant future 
(year 2020), autonomous robots could be used to assemble detailed 
structures in space. In these applications noncontact sensing could 
involve ranging and identification, as well as motion monitori ng. 

In to areas of inspection , maintenance, and repair , many 
tasks must be performed. Inspection will monitoring 

fttigue, failures, flaws, meteorite damage, structural vibrations, and 
environmental parameters. The maintenance of i space system can 
involve to servicing of avionics, thermal system, brakes, landing 
gear, TV cameras, recorders, printers, actuators, fluid coolant 
heaters, tonuo s ta ts, circuit breakers, switches, power systems, and 
aging composite materials. Initially, toy could be done using 
Mobile RMS (MRMS) on to Space Station (Figure 2). In to dis- 
tant fiiture, to inspection/mrintenanoe/repair functions would be 
highly au t o m a t ed and continuous (Figure 3). This would mean that 
to robots have to be versatile and adaptive. The virion sensors for 
these operations should include a foil range of sensing which is 
independent of nature] light Many servicing functions such at 
refurbishment, resupply, refueling, cleaning, and storing human 
tefose and debris, and replacement of batteries could be automated 
with m ini m u m virion, involving only the video sensors. However, 
most of these tasks require doclring/berthing which, when 
automated, could be made fast and stable. This operation can be 
made economical by choosing a vision system which is accurate and 
very fast 


A major thrust for utilization of space environment is 
manufacturing. Many processes have been envisioned for micro-g 
stable environment, which can yiueid unique products hard or 
ikmpottible to manufacture on earth. These processes are long 
duration, need relatively contamination-free environment, can be 
dangerous to humans, and/or require disturbance-free surroundings. 
Thus, it is either mandatory or desirable that 
sy stem/robot* provide to servicing and 0 f th e se facili- 

ties. General Electric, in a recent review [3], recommended that to 
manufacturing fodlity to able to perform slicing, polishing, clean- 
ing, sawing, separation, ion implant, photo resist, anne aling, E- 
beaxn direct write, and reactive ion itch. One opportunity is to pro- 
duce computer chips from hazardous material gallium arsenide for 
•uper computer systems. The critical limitation on to virion sen- 
son imposed by to manufacturing task is small size. In most cases 
to need for vision will arise for human monitoring purposes. Thus, 
a distributed system spread across to laboratory might be required 
to cover all areas. For to robotic end-effector, in addition to small - 
neat, a highly accurate and high resolution system will be required. 

Ftir to direct control teUoperator systems, the primary func- 
tion of robotic virion is to provide information about to position of 
to object relative to to system’s effector (Figure 4). In a direct 
controlled teleoperetor system, some subfunctions may be allocated 
to to h u m a n . Furthermore, the object identification can be 
delegated to to hu m a n After to description is given by to human 
[4], foe mach ine generates a model from to observed video image. 
This model uses a generic description from to data base. The pat- 
tern parameters derived from the video-based model provide to 
necessary information to generate to end-effector position com- 
mand (Figure 5). 

In a goal-directed teUoperator system, reliability and flexibil- 
ity have to be provided, with which it searches, identifies, and 
l o c at es pons baaed on existing data base. Tt* CAD/CAM type 
information and directives are provided within to teleoperazor sys- 
tem. In to absence of to virion system, to required object 
identification and position data must be entered manually. In a 
higher level goal-directed teleoperator system, to operator supplies 
goals. These goals are then broken into tasks by to knowledge 
based CAD/GAM information regarding to processes involved. 
Complex tasks are broken into simpler commands which are tzxe- 
cuted using known rules and sequences. In such a system, a con- 
tinuous data flow must be maintained from to vision sensors. The 
location data should not only identify to object but also to 
porition/motion parameters of to end-effector at each inarew of 
time or successive step in to accomplishment of to task. 

In general, for to autonomous robotic systems , three levels of 
information are n eeded . These levels pertain to to scene/world in 
which to objects are located, to objects themselves, and specific 
pans of to objects. In most tasks an envelope of parameters can be 
preprogrammed into to system. For example, in Av-Jnng 
berthing applications to robotic virion/senring may be needed 
within a cone of, say, thirty degrees to a of 50 meters. 

Beyond this spatial zone, a radar system may be used for to 
tncking/motion monitoring. The levels of information depend on 
to application involved. As an example in to satellite servicing 
area, the virion/sensing system may have to provide to necessary 
information to guide a robot/astrobot to a parri^i^r ares, say, an 
antenna feed. The satellite could be rotating and translating simul- 
taneously. Furthermore, to antenna could be gimbaling with a cer- 
tain motion. In this scenario to data would not only include a 3D 
dynamic description of to target/object but also its position and 
rotational parameters with respect to to satellite. To accomplish 
this, algorithms are needed for to parameter estimation. The 
virion/sensing instrumental ton in this case would not only involve 
fixed field of view video systems, but laser/millimeter wave radars 
which could be slaved to to antenna feed motion. Doppler signal 
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processing at microwave or optical frequencies can be used to sense 
moving pans within a scene. The implication on the vision/sensing 
systems is clear, several sensors axe needed to complete the basic 
information needed for an autonomous robot G early, for space 
applications, the size, speed, and weight parameters are of 
paramount importance. 

Autonomous robot performance depends crucially on the 
vision capabilities. In certain operations, humans can be surpassed 
by robots based on the memory and vision. The use of color and 
polarization in future will become increasingly important, since 
variations in these parameters are related to certain scene features. 
General application areas for machine vision are given in Table 3 
[51. 


Measurement 

Category 


Sample 

Application 


Thrust 


Location 

Identification 

Recognition 

Inspection 

Gauging 

Counting 

Motion 


Robotics/Ranging 

Assembly/Inspection 

Sorting/Picking 

Quality 

Industrial Control 

Inventory 

Ranging 


Where 

What 

Which 

How Good/ Accurate 
How Large/Much 
How Many 
When/Where 


Table 3. General Application Categories for Machine Vision 


The vision extension to shadowed and occluded regions is 
important in many applications. The illumination intensity varia- 
tions along with shadows can, in certain applications, be used to 
determine object shape, as well as an estimate for the relative 
motion between the camera and the object Mathematical models, 
coupled with real-time imagery, can be used to derive these parame- 
ters. Structured multi- spectral lights can be used to derive the 3-D 
description of the t ar get Associated with the sensory data is the 
need for compiler architectures [6] which provide high-speed pro- 
cessing, parallel computations/algorithma, associative memories, 
and intelligence. Use transfer and reduction of the sensor data can 
be facilitated with the implementation of a communications subsys- 
tem. 

The vacuum of space makes the scattering of Light more 
dependent on the surface geometry and properties. The usual 
illumination caused by diffraction associated with the reflection of 
light from objects on earth is negligible. This phenomenon yields 
pronounced shadows and specular points for space -acquired images. 
Another detail that should be taken into account is the thermal pro- 
tection system (e.g., thermal blanket) which obscures direct visual 
detail of electronic and other systems requiring such protection. In 
addition to this, occluded areas also exist in the scenes to be sur- 
veyed. In view of these environmental constraints, space 
vition/seniing systems shook! provide a mode independent of sun- 
light To achieve this independence along with penetration, struc- 
tured optical ill uminati on and microwave sensors might be con- 
sidered. For this multiple source illumination, the wavelength, 
intensity, polarization, field of view, and angles of i n ci dence must 
be chosen for a particular scene parameter (Figure 6). These imple- 
mentations will ensure appropriate enchancement of texture and 
dependence on scene color necessary for certain robotic tasks. 

The present vision/sensing systems have several serious limi- 
tations which result in these being unable to provide 
data/ parameter needed for autonomous robots to have 
perception/reasoning. These limi tations include problems with sha- 
dows, occlusions, high contrast, low resolution, 2-D models, rigid 
bodies, and viewing from a fixed viewpoint In addition to these, 
cost-effectiveness, speed, small size, lightweight, high reliability 
and flexibility, a a! ease of operation must be considered. 


///. Non-Contact Sensors for Data Collection 

In robotic vision systems electromagnetic energy emitted or 
reflected by an object is received, converted/transduced, and finally 
processed. Transducing usually converts^ecefved energy into 
electrical signals, and processing transforms those signals into 
parameters needed for perception. This perception then allows the 
robot to execute a function. Processing can be accomplished in two 
steps: that of preprocessing , which involves 

compensation/corrections for various hardware characteristics, 
enhancement, selection, and decoding/decompression; and that of 
analysis! interpretation . The latter step is carried out with specific 
algorithms to extract die required informatioa 

In the initial vision systems, NASA anticipates the use of 
stereo televisions for label/feature baaed object recognition [7]. 
NASA 1 ! television progr am from Apollo through the Space Shuttle 
programs has been one of high crew and ground participation and 
control. The sophistication of work to be performed either by teleo- 
perated or autonomous robots points out several limitations of these 
systems. These limitations surfaced during the Solar Max Satellite 
repair, when shadowed surface could not be approached, and the 
movement and grappling of the Solar Max were severely restricted 
because of limitations with the manual television camera light level 
controls. Several modifications are now being developed at the 
Johnson Space Center (JSC) for future space television systems. 
These include predictive auto focusing, programmable predictive 
scene control with auto zoom, gamma, and iris, automated or voice 
control pan , dlt , and pointing capability . Breadboards of these sys- 
tems are currently under evaluation at JSC [Figures 7, 8]. The 
development of illumination systems for enhancing image qualiry 
and extracting shape from shading is also being pursued at JSC 
[Figure 9]. This development is guided by analytical 
methods/ algorithms discussed in Section IV. 

One of the problems in accurate recognition of objects in 
space is the Earth background. Spectrum measurements taken at 
NASA/JSC [Figure 10] show absorption bands in the near-infrared 
region. The implementation for the video utilized charge injection 
devices (CID) to avoid image blooming with optical load compared 
to charge coupled devices (CCD). The CID sensor (GE TN2505 
camera) has adequate responsivity at the 0.94 \xm wavelength. This 
wavelength provides a water absorption notch. Thus, in space this 
sensor will be earth blind. The calculated attenuation in absorption 
for a 0.04 pm band is 21.6 dB. The intended use for this sensor in 
space is pointing and following an object in motion. For this pur- 
pose an 8-bit microprocessor is used to control the adaptive system 
[Figure 11]. This unit has only 2 kilobytes of random access 
memory (RAM) and 2 kilobytes of read only memory (ROM) which 
is switchable to 4 kilobytes [8]. The system parameters which can 
be varied include camera sensitivity, digital filtering, video thres- 
hold, and zoom lens setting. The incoming infrared enhanced video 
is compared to an a djustable thr eshold, which is varied under the 
microprocessor control, to generate a aerial data stream. An ade- 
quate digital filter is used to retro ve high frequencies from this data 
stream. The data is then applied to two counters: one counter accu- 
mulates the number of pixels above the threshold, which provides a 
measure of the total area of the object; the other counter is used to 
count only on the left tide of the object in the scene and provides 
the area oc the left tide of the scanned area. In a similar way the 
area is also measured in the top half of the scanned area. These 
areas are compared to Minfin die object in the center of the field 
of view. This procedure then results in providing commands to the 
pan and tilt unit The parameters that affect the dynamic charac- 
teristics of die tracking system can be selected for optimum stability 
and response. 

Another advancement in spaoe TV operation incorporated at 
JSC is the voice control [9]. The initial Voice Control System 
(VCS) has the following performance parameters: (1) isolated and 




continuous speech recognition, (2)200 word vocabulary (it 1.2 
sec/word), (3) two person support (3rd optional capability), 

(4) user-trained (dependent recognition) for high accuracy, And 

(5) symaxing capability. This system [Figure 12] will be tested 
onboard Shuttle and has been made to connect with the Shuttle 
Audio Distribution System and existing headsets. The system 
allows complete hands-off control of CCTV functions including: 
(1) monitor selection, (2) camera selection, and (3) pan, tilt, focus, 
iris, zoom, and scene track. Future use of voice for TV automation 
has been projected for the Extra- Vehicular Activity (EVA) 
astronaut. In this application the astronaut can ask for Heads Up 
Display (HUD) of vital data from the Shuttle computers. These data 
can i n clud e system parameters, orbital parameters/! ocation, system 
status, and particular subsystem data [Figure 13]. In the robotics 
application, these systems can be incorporated using synthetic voice 

The need for video system data to be able to interface with 
digital processon/computers has given impetus to digital TV tech- 
nology. For greater reliability, solid state imagers with sensor sizes 
512 x 512 and 1024 x 1024 are being designed. Furthermore, 
recognition/preprocesting algorithms can be implemented on (he 
same electronics chips making the size of these video imagers 
small , These technology innovations are also timed at high resolu- 
tion systems. As this technology is moving forward rapidly, the 
need for handling and transmission of high data rates is becoming 
obvious. For a video system at 5 MHz baseband, sn eight bit digiti- 
zation would generate SO MBPS data stream. For color TV imple- 
mentation and multiple systems, this bit rate can multiply 
significantly. For real time processing of this data . compression 
techniques have been proposed. These compression algorithms can 
be hardware implemented using VLSI and VHS1C circuitry. Tire 
algorithms must be automatic and transparent to users, not destroy 
or d i s c a r d any relevant information, and compress and decompress 
data at speeds significantly higher than associated device data 
transfer speeds. 

Fourier optical processing offers a method of high speed paral- 
lel processing of data needed to support automation ami robotics 
applications [Figure 14]. The inherently parallel nature of optical 
information processing, coupled with foe easy and natural optical 
Fourier transform and foe programmable masks, can obviate numer- 
ical processing for many applications. The ***&* are used to modu- 
late foe optical Fourier transform of an input scene [Figure 14]; an 
optical recransform then allows direct detection of, say, foe 
mathematical correlation between the viewed scene and foe refer- 
ence image, the mask of which is placed at the location of the opti- 
cal Fourier transform. In this manner any image computation that 
can be cast in the form of correlation or conv olution be tween o bject 
and reference images can be approached with optical processing 
Programmable masks are yet in the rudimentary state. Texas Instru- 
ments, under joist sponsorship by NASA/JSC and foe Army Missile 
Command, has fabricated a Deformable Minor Device (DMD) 
which is under test and further development at this time. The per- 
formance degradation caused by optics and diffraction/scanering 
effects is being studied, and devices/techniques to compensate these 
effects are being developed. Another area of extention is filtering 
with a controlled amount of scale and rotation invariance for use in 
the comrol/docking applications. Furthermore, Johnson Space 
Center hat designed a high-speed special-purpose geometrical 
remapping image processor. The device is in manufacture by Texas 
Instruments; delivery is expected in Fall 1987. Insensiti vity to scale 
and rotation of a viewed object will be the result of one form of 
mapping, allowing one to use only spatial displacement in tracking 
the object The elimination of scale and rotation dependence greatly 
increases the speed of the vision system for its use in a control net- 
work. Experiments with various mappings will result in the design 
of VLSI cameras whose receptor patterns are best suited to drive a 


subsequent optical correlator. 

For foe 3-D vision, laser scanning devices providing range tt 
each point of the target are being explored at NASA/JSC. A matrix 
of ingles/ position and ranges within the field of view provides 
depth/height profile of the object. These measurements can be com- 
bined with the video reflectance measurements to yield a 3-D real 
time visual definition of foe target Two technology implementa- 
tions of this solid state laser vision device currently available are 
those using mechanical motion of the mirrored surfaces, and those 
that involve an inertialess change in foe optical properties of a tran- 
sparent medium. The latter class includes diffraction of light from 
an acoustically generated periodic structure. Phased- array solid 
state scanning devices are currently being developed These devices 
can provide fast, accurate, and lightweight laser vision. The data 
from laaer vision devices can also be used for automatic zoom/focus 
control of video systems. 

The laaer vision measurements are dependent on foe intensity 
of reflected radiation. If coherent radiation is used in order to gen- 
erate an of the object information from the amplitudes, as 
well as the phases of the scattered radiation, a 3-D reconstruction of 
the object can be made. Such devices are known as holographic 
devices. The source of coherent radiation can be a solid state laser 
fojr can in principle provide a resolution of the order of about 1 pm. 
Part of the radiated beam is deflected toward foe detector [Figure 
15], where it interferes with the backsctttered light from the object 
The hologram can then be generated using known reproduction 
processes, or a t hree dimensional description of the object such as a 
Fast Fourier Transform (FFT). Several applications of holographic 
scanners have been discussed by Sincerbox [10]. Some of these are 
directly applicable to space robotics systems. 

Microwave systems have been used to detect relative speed of 
objects and their range in many applications. Their use in space 
robotics applications is being studied at NASA/JSC. In particular, 
millimeter wave radars provide attractive performance parameters, 
in addition to their mall size. The possibility of broader beam than 
laser systems mafcea these sensors attractive for initial acquisition of 
moving objects. A radar at 100 GHz has been developed at 
NASA/JSC [11], This particular system is for use on a Man 
Maneuverable Unit (MMU) to provide relative range and velocity to 
foe object [Figure 16]. The radar is designed to operate over the 
range of speeds from 0.1 to 10 fps. This type of radar, operating at 
several carrier frequencies, can be used to measure backscanering 
coefficients for various polarization combinations. These 
coefficients are object structure dependent There is also possibility 
of penetration through thermal protection and obscuration caused by 
nonmetallic objects. This data can be utilized in an iterative manner 
with that of the video systems to provide scene 
definition/paiameters in certain shuationa/scenerios. 

IV. Alforithms for Information Processing 

Robotic vision, as it is generally applied at present, is mainly 
based on video-acquired data. The acquisition of this data takes 
place in the form of time sequences of images obtained by one or 
more vidicon or solid state cameras. In this process, A/D conver- 
sion, image arithmetic (e.g., addition/ subtraction of two s ucce ss i ve 
images), and formation of pixel space constitute steps usually taken. 
Both 8-bit gray level and 8x8x8 bit color systems are commonly 
u re* However, tire resolution and linearity achieved by the state- 
of-the-art came ras are somewhat limited. To compensate for this, 
many cameras are needed in some industrial applications. For 
example, one sheet metal tide rail inspection system uses over three 
hundred cameras [5]. The corresponding images are patched 
together into a super pixel space. 

Image acquisition is followed by several stages of informa- 
tion processing, namely (i) image preprocessing (filtering, 
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restoration, and enhancement), (ii) feature extraction (e.g., extrac- 
tion of centroids, perimeters, and areas of patches in the image, 
Gaussian curvature from the shading on these patches, min/max dis- 
tances, edges, comers); (iii) Object recognition/ classification ; (iv) 
estimation of motion parameters of these objects (position, velocity, 
attitude, and attitude rate), and (v) image understanding! scene 
interpretation 

Active research in vision information processing is being pur- 
sued at various centers in this country, and has been widely docu- 
mented in the literature (see, e.g., [12H17), [30] and the references 
therein). In what follows, we highlight the bask philosophy and 
thrust of k >ttw of die algorithmic work currently in progress at Rice 
University. This work is being carried out at the Computer Vision 
Research Laboratory. The equipment used in this research includes 
a system of solid stats cameras such as the Hitachi KP232 with 
accessories, connected to Series 100 Imaging Technology circuit 
boards, and horn these to a SUN Microsystems 3/160 color works- 
tation and a VAX 11/750 minicomputer with several peripherals. 
Cardboard and plastic models of space objects, such as satellites, the 
shuttle, and pans of the space station (see Fig. 17) are being used in 
a studio environment, under controlled illumination conditions, for 
generating the laboratory data needed to provide insights for algo- 
rithmic development. The final testing of the developed algorithms 
will be carried out on the computer vision testbed (Fig. 9) at the 
N AS A/Johnson Space Center mentioned earlier. 

The ™in points of view on which the algorithmic work at 
Rice is based as well as some of the developing results are now 
listed below; 

(i) Model-Based Approach: In most space scenarios as well as 
in industrial environments, die objects being viewed are unlabeled 
objects from a library of known arid precisely defined objects. So in 
our algorithms work, we assume this to be the case, and attempt to 
capitalize on it to reduce die hardware requirements and the compu- 
tational effort needed to execute many of the 3D vision functions 
executed by a robot, as indicated in (ii),(v) and (vi) below. 

(ii) The FlAG and M1AG Representations of 3D Objects: 
We defii* a rigid 3D object q by the surface SW) bounding it 
We i tpmw 5 ( r\) to be piecewise smooch and satisfying conditions 
which permit it to be represented as a union of smooth surface 
patches or "faces* Ft, Fi % F* , (See Fig. 18 .) The boundary 

of each face F, consists of a closed contour C, , the union C% 

of all such contours constituting what we call the m 3D wireframe? 
of the object r \ . This allows the symbolic representation of the 
object ii by an attributed graph GW) (which wraps around S(n) 
\ the nodes A/,,i a .and the links ty,J m t 

of GW) the connectivity between adjoining faces F; and Fj . Let 
an m-vector /‘ « c »/ (/{, ...,/i) represei* a set of attributes 
(features) associated with d* face F, which are invariant under 
3D translation, scaling, and rotation. Examples of such /' are the 
sets of numbers expressing the Gaussian curvature or mean curva- 
ture of Fi . We call the attributed graph 6 (ri) obtained from 
G(n) by assigning to die nodes N* , i » 1, * , respectively, the 
attribute vectors 7 1 , die FLAG (Feature- In variants/ Attributed - 
Graph) representation of the object q. A special case of this 
representation is the MlAG (Moment-Invariants/ Attnbuted^raph) 
representation of polyhedral objects proposed in [18], in which /' 
constitutes a sec of 2D moment invariants of F, (these being 
invariant with regard to 3D translation, sc a ling , and translation). 
(See Figure 19). 

(iii) Wireframe Extraction: For the utilization of the FIAG 
and MIAG representations (see (v) and (vi)), it is necessary to 
extract the 2D-wireframe W(f(r\)) mW <f) of die image / of 
x \ . W(q) is up of the set of edges in / . In order to extract 
WW) . * number of operations have to be performed on the raw 


image data. These are Gaussian 2D filtering (to remove noise), 
edge detection by application of a (3 x 3) Sobe! operator followed 
by thresholding (horizontal and vertical), median filtering (to delete 
spurious edges), edge thinning, and model -based reconstruction to 
recover missing segments in W (q). 

(iv) Wireframe Labeling and! a Attributed Graph Represen- 
tation: W (tj) is labeled by an algorithm [19] in which each mesh 
of Mi of WW) i* traversed clockwise and the outer loop counter- 
clockwise. This algorithm also determines the adjacency oT the 
regions R l enclosed by the meshes M t . The regions /?, defined 
by W (rj) correspond respectively to the visible faces F t on S (T]). 
Also, in the polyhedral cate, the moment invariants of R t are the 
same as those of Fj . Thus, in this case, a MIAG representation 
C if ) can be obtained for f , based on W(f ), and cleariy 6 ( f ) 
is a subgraph of (5 (q) . 

(v) 3D Object Identification from a Single Image. It follows 
from the preceding that the identification of an unlabeled object t\ 
from its image / is achieved by obtaining <5{f ) and matching 

<5{f) to a subgraph of one of the graphs 6 W*)*i * 1 N, ina 

library of models by a subgraph isomorphism algorithm [18]. 

(vi) Motion Parameter Estimation with a Single Camera. 
Using appropriate camera calibration, all the motion parameters, 
except for a scaling factor, can be measured by means of a single 
high precision camera. For this purpose, there are basically rwo 
model-based methods available: One, based on the contraction of 
the moment tensors of a surface patch of the model and its image, 
determines the attitudes vector 0 (raw, pitch, and yaw) and attitude 
velocity 0. (See [18] and [20] for details.) The other, based on the 
correspondence (assumed known) of eight points on the image / 
and eight points on the model tj' (assumed located and oriented in 
a standard position), yields all the motion parameters (position, 
velocity, attitude, and attitude vector) except for a scaling parame- 
ter. 'Hris second method has bees extensively discussed by 
Longuet-Higgins[21] P Huang [22], Haralick[23] and others. Our 
contribution to the second approach is the ability to establish the 
point correspondence needed for it by obtaining G (J ) from / and 
matching Gif) to a subgraph of G W) Along the lines indicated 
earlier. 

(vi) Application to Stereovision: The approach just men- 
tioned can ato be used to establish the point correspondence needed 
for stereovision. . , 

The model-based approach proposed in the above paragraphs, 
even though not fully investigated, appears to offer two advantages: 
First, it reduces the hardware by using only a single camera for 
identification and for most of the morion parameter estimation of a 
moving object Secondly, the attributed graph representation and 
subgraph matching proposed here avoids the computational effort 
and problems inherent in (he probabfisric search associated with the 
matching at the feature level [30). 

(vii) Illumination: Shape from shading algorithms are 
expected to play a significant role in the analysis of space scenes 
because light scattering from space objects strongly depends on 
their surface due to the prevailing vacuum. The dianuaerist ic snip 
expansion methods [16*24] have several shortcomings, including 
sensitivity to measurement noise and s tende n cy of adjacent chsra c- 
teristic strips to cross over each other, due to accumulation of small 
numerical errors. Finally, the procedure is not amenable to imple- 
mentation in parallel form. The variational method [24] that uses an 
object's occluding boundaries as cues to the recovery of its shape 
from shading alleviates these limitations. The blending of concepts 
from variations] calculus with those from the best approximation 
theory can lead to spline-based solutions for the gradient functions 
determining the local surface shape orientation, as obtained in [25]. 



(vM) Surface Reconstruction: Algorithms for surface recon- 
struction from sparse range data can be of value in providing vision 
to teleoperators. The reconstruction of a surface can also be used as 
an intermediate step in extracting surface features for machine 
recognition. Mathematical analysis [26], differential geometry in 
particular, as well as formal language theory (in tynuctic/semantic 
shape description) can play a significant role in surface reconstruc- 
tion algorithms as indicated in [27]. 

(ix) Sensor Fusion The fusion of data horn more than one 
type of sensors can be of value when any one sensor cannot capture 
the full information needed. Algorithms for fusion of microwave 
and video data introduce significantly the wave equation scattering 
theory into robot vision [28]. 

(x) High level processing : Apart from techniques based on 
geometric models, which have been dis cu ssed in the previous sec- 
tion, current efforts on machine vision explore A1 techniques. 
These are based on symbolic scene description. Such a description 
enables one to transform die image into knowledge which can be 
utilized in the conditioning of the action and the behavior of a robot 
Thus scene understanding can be viewed as a reasoning process 
based on visual data. This process can consist of common sense 
reasoning , for which the knowledge is wide and shallow, and expert 
reasoning, based on die narrow and deep knowledge of experts. A 
recent effort has been directed toward a frame-based approach to 
model common sense reasoning [29]. Object-oriented programming 
is particularly suited for implementing such an approach. In addi- 
tion, parallelizing the frame manipulation in image undemanding 
appears to be promising for a real-time solution and it is under 
investigation. 

V. Proposed Future Developments 

As was mentioned earlier, the interaction of natural light with 
die objects in space has to be accounted for in the algorithms. 
Furthermore, methods for artificial lighting have to be developed 
which can provide structured (known distribution) light across the 
object The pronounced shadows and specular points due to the 
vacuum and smooth pans of the object, provide a large dynamic 
range of the reflected/scatxered signal. The intensity changes can be 
in the 10* range. The addition of artidfial illumination provides 
the opportunity to control intensity, wavelength, polarization, and 
orientation with attendant advantages of increased recognition and 
shape estimation capabilities. Additionally, color will be another 
discriminant involved in the algorithms. Analytical studies in these 
areas should lead to the design of illumination systems for space 
applications. 

The use of laser vision and microwave scattering instruments 
creates another area of fature development Fast 
•canning/holographic lasers provide a depth perception of objects, 
which is quite complex. This depth data can be utilized to itera- 
tively provide a 3-D image of the object by weighting video- 
acquired image data appropriately. These weights will depend on 
the surface curvatures as they project in the inc id ence direction of 
the laser b*mm Both empirical and analytical studies are needed. 
The microwave hackscattering can provide another independent set 
of data. The shape of certain objects can be directly de du c e d from 
this data. In many inspection tasks in which a nonmetallic shielding 
has obscured the view, such sensing will be mandatory. In other 
situations the microwave data can be iteratively used with dial of 
TV to arrive at a more definitive description of the object At the 
expense of complexity, doppler processing of microwave data can 
yield the shape of a distant object The advantages of such a vision 
are that it is independent of sunlight and it provides a direct m easure 
of range and relative velocity of the object 

Another area of endeavor should be near-field sensing. In this 
mode a microwave sensor can be very close to the object A sharp 


pulse transmitted yields a unique description of the object This 
time demain refleexometery is evolving rapidly. Another mode of 
the system can utilize reflectivity data in the near-field. These tech- 
niques have not beer explored for the robotic vision applications. 

Finally, further research and development is needed in the area 
of multisensor coordination and fusion. The recognition algorithms 
are to be extended to include interrelating data from several cam- 
eras, laser tcannen/holographic systems, and microwave sensors. 
These algorithms should include motion, rotation, and object 
changes as functions of space and time. In addition to this, 
"environmental" d*** , pertaining to the events/objects and their 
status, has to be included These aspects, along with rational 
models, incorporate artificial intelligence techniques in the scene 
analysis, driven by common-tense systems and expert systems. 

V7. Conclusions 

This paper is timed at providing a review of some of the 
efforts in progress at NASA/JSC and Rice University. The design 
and development of a vision system for space applications needs 
several considerations which make them different compared to 
those used in ground applications. The concerns for space-unique 
vision systems have been elaborated Several efforts which need to 
be undertaken have been discussed Considerable work has to be 
accomplished in order to provide robust, lightweight, real-time 
(fast), and size vision systems for specific space applications. 

ML AcknowUdfment 

The authors wish to thank Mr. R.S. Sawyer of the 
NASA/Johnson Space Center for his encouragement and continued 
support of the research reported in this paper. 

MIL References 

1. "Advancing Automation and Robotics Technology for the 
Space Station and for the U.S. Economy," NASA Technical 
Memorandum 87566, VoL I, Executive Overview, VoL II, A 
Technical Report, March 1985. 

2. Teaar, Delbert, "An Assessment of the Development and 
Application Potential for Robots to Support Space Station 
Operations," College of Engineering, The University of Texas, 
September 1985. 

3. "Space Station Automation Study, Automation Requirements 

General Electric Company Report for NASA Contract No. 
NAS 5-25 182, July 1984. 

4. Meimtl, AJ., and Ralph W. Will, "Space Teleoperation 
Research," NASA-Langley Research Center Report presented 
at the American Nuclear Society Executive Conference, Fine 
Mountain, Georgia, April 21-24, 1985. 

5. Murray, Lawrence A., The Present and Future State of 
Machine Vision Industry," NASA Automsrion and Robotics 
Workshop, Johnson Space Center, January 21, 1986. 

6. CUffow, Donald L., "Sensing for Automation and Robotics," 
A White Paper for the NASA/OAST Sensor Working Group, 
August 1985. 

7. Holcomb, L3., R. Larsen, and M. Momemerio, The NASA 
Automation and Robotics Research Program," AIAA/NASA 
Symposium on Automation, Robotics, and Advanced Comput- 
ing for the National Space Program, Washington, D.C., Sep- 
tember 4-5, 1985. 

8. Lipoma, P.C. and DJ, Walton, "A Two-Dimensional Near- 
Infrared Tracking System," SPIE’i Cambridge Symposium on 
Optical and Electro-Optical Engineering, Cambridge, Mass., 
September 15-20, 1985. 



9. Jordan, William T., "Voice Controlled Closed Circuit Televi- 
sion for the Space Shuttle Orbiter" The Official Proceedings 
of Speech Tech. ’85, New York, April 22-24, 1985. 

10. Marshall, Gerald F. (Ed.), Laaer Beam Scanning, Marcel 
Deleter, Inc., New York, 1985. 

11. Uchtenberg, C.L., Man Maneuverable Unit MiUimeter-Waye 
(100 GHz) Developmental Radar - Design and Test Results 
Report, NASA/Johnson Space Center, Houston, Texas, 1986. 

12. Rosenfeld, A., and A. Kak, Digital Picture Processing, VoU. I 

& II, Academic Press, NY, 1982. ” 

13. Marr, D„ Vision, W.H. Freeman Press, San Francisco, CA, 
1982. 

14. Cohen, PJL, and A. Feigenbaum, The Handbook of Artificial 
Intelligence, Vol. 3, Herns tech Press, Stanford, CA, 1982. 

15. Ballard, DJL, and CM. Brown, Computer Vision, Prentice- 
Hall, Englewood Cliffs, NJ, 1982. 

16. Horn, B.KP., Robot Vision, McGraw-Hill, NY, 1986. 

17. Winston, P., Artificial Intelligence, Addison- Wesley, Reading, 
Mass., 1984. 

18. Bamieh, B., and RJP. de Figueiredo, "A General Moment- 
Invariants/ Attributed -Graph Method for 3D Object Recogni- 
tion from a Single Image," IEEE Journal of Robotics and 
Automation, vol. RA-2, no. 1, pp. 31-41, March 1986. 

19. de Figueiredo, RJ.P., B.A. Bamieh, S. Fotedar, E, Hack, K. 
Trahan, and C. Wu, "Demonstration of a Methodology for 
Machine Recognition and Attitude Determination of a 3D 
Object from a Single TV Picture Frame", Technical Report 
EES 5 20, Dept of Electrical and Computer Engineering, Rice 
University, December 1985. 

20. Markandey,V„ H. Tagare, and RJP. de Figueiredo, "A Tech- 
nique for 3D Robot Vision for Space Applications", Proc. of 
Space Telerobotics Workshop, JPL, Pasadena, CA, Jan 20-22, 
1987. 

21. Longuet-Higgins,H.C., " A Computer Algorithm for Recon- 
structing a Scene from Two Projections," Nature, vol. 293, pp. 
133-135. Sept’81. 

22. T*ai,R.Y., and T. S. Huang," Uniqueness and Estimation of 
Three-Dimensional Motion Parameters of Rigid Objects with 
Curved Surfaces,” I FFF Trans. Pattern Anal. Machine ImelL, 
vol. PAMI-6, No. 1, January 1984, pp. 13-27. 

23. Zhuang.X., T. S. Huang and R. M. Hanlick," From Two- View 
Motion Equations to Three-Dimensional Motion Parameters 
and Surface Structure : A Direct and Stable Algorithm,” Proc. 
tpff int Conf. on Robotics and Automation, San Francisco, 
California, April 7-10, 19S6, pp. 621-626. 

24. Horn, BJCP. and MJ.Brooks, The Variational Approach to 

Shape from Shading," MIT AI Laboratory Memo 813, March 
1985. 

25. de Figueiredo, RJP. and V. Marktndey, "Recovering Shape 
of Space Objects from the Shadowing Caused by Illumina- 
tion," Tech. Report EE-8608, BCE, Rice University, Sep- 
tember 1986. 

26. Kehtaroavaz, N. and RJP. de Figueiredo, "A Novel Surface 
Reconstruction Framework from 3D Contours," Proc. of 
SWE's Cambridge Symposium on Advances in Intelligent 
Robotic Systems, Cambridge, MA, October 1986. 

27. Gregory, J.A., ed. "The Mathematics of Surfaces", The Insti- 
tute of Mathematics and its Applications Conference Senes, 
September 1984. 

28. Shaw, S. W., K. Kristen, and R. J. P. de Figueiredo, 
"Microwave ami Video Sensor Fusion for Shape Extraction of 


Space Objects" Rice University Report EE-8706 

29. de Figueiredo, RJP. and Kuanshen H. Wang, "An ’Evolving 
Frame’ Approach to Learning with Application to Adaptive 
Navigation", 1986 International Conference on Systems, Man, 
and Cybernetics, Atlanta, GA, October 1986. 

30. Wilcox, B., D.B.Gennery, B.Bon, and TXitwin "Real-Time 
Model-Based Vision System for Object Acquisition and 
Tricking", SPTE International Conference, Los Angeles, 1987. 



FlgJ Conceptual Space Station Automation/Robotics 



F1g.4 Teleoperator Concepts for Shuttle Operations 


s 




Fig.l Functional Elements of Teleoperator/Autonomous System 



fig J5 Direct Controlled TeUoperator Vision 


SCENE DEFINITION TECHNIQUE 

for video system 



YigA Struc tur ed Illumination tor Robotic VWon/Souing 



Fi^lO Spectrum analysis of energy reaching the Earth 



Flg.ll Video Pointing System 



















"* ^ ^ "COMM STATUS" 

Flg.12 Voice Control Syrtetn for Shuttle TV 


Flg.13 Conceptual EVA TV Voice Command Syttem 


10 









Flg.14 Programmable Mask Technology 



fYJAQA 

■ IV Idv « Space t &rarts r mo n 


naanr W r 'T56# 



with MMU Radar 
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